HEMP: High-order entropy minimization for neural network compression

We formulate the entropy of a quantized artificial neural network as differentiable function that can be plugged regularization term into cost minimized by gradient descent. Our formulation scales efficiently beyond first order and is agnostic quantization scheme. The then trained to minimize parameters, so they optimally compressed via coding. experiment with our at quantizing compressing well-known architectures over multiple datasets. approach compares favorably similar methods, enjoying benefits higher estimate, showing flexibility towards non-uniform (we use Lloyd-max quantization), scalability any efficiency in terms compression. show HEMP able work synergy other approaches aiming pruning or model itself, delivering significant storage size compressibility without harming model's performance.